Papers with proximal policy optimization
TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use (2025.findings-emnlp)
Copied to clipboard
Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, Zhengyin Du
| Challenge: | a new approach to training large language models (LLMs) overlooks task-specific characteristics in tool use, leading to performance bottlenecks. |
| Approach: | They propose a task-feature-based framework that mitigates the effects of suboptimal training data . they use a dataset to train large-scale LLMs and a reward mechanism tailored to error categories . |
| Outcome: | The proposed framework matches or surpasses open- and closed-source LLMs in tool-use performance using only 1,217 training data points. |
P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Contemporary approaches to generate tabular data are limited due to the lack of external knowledge. |
| Approach: | They propose to use proximal policy optimization to apply GANs and fine-tune Large Language Models to enhance the probability distribution of tabular features. |
| Outcome: | The proposed method improves accuracy of GANs and LLMs over state-of-the-art over three real-world datasets. |
Finding a Needle in the Adversarial Haystack: A Targeted Paraphrasing Approach For Uncovering Edge Cases with Minimal Distribution Distortion (2024.eacl-long)
Copied to clipboard
| Challenge: | Adversarial attacks against Language models (LMs) are a significant concern. |
| Approach: | They propose an approach to automatically learn a policy to generate challenging examples that improve the model’s performance. |
| Outcome: | The proposed approach outperforms baselines and exhibits generalizability across classifiers and datasets. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Probabilistic Robustness for Data Filtering (2023.eacl-main)
Copied to clipboard
| Challenge: | Modern machine learning works with massive amounts of data on a range of tasks like language modeling, object detection, and data mining. |
| Approach: | They propose a probabilistic robustness rewarded data optimization approach to enhance the model's generalization power by selecting training data that optimizes probabilistic metrics. |
| Outcome: | The proposed approach achieves +17.2% increase of accuracy and -28.05 decrease of perplexity on unknown-domain test sets. |
Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language models (LLMs) are trained on vast amounts of data, including sensitive information that poses a risk to personal privacy if exposed. |
| Approach: | They propose a novel unlearning approach that utilizes an efficient reinforcement learning feedback loop via proximal policy optimization to incentivize the LLMs to learn a paraphrasing policy to unlearn the pre-training data. |
| Outcome: | The proposed approach surpasses strong baselines and state-of-the-art methods in terms of its ability to generalize and strike a balance between privacy and LLM performance. |
EROS:Entity-Driven Controlled Policy Document Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | a privacy policy is a crucial component of any organization that allows it to legally collect, process, store, and/or distribute personal data. |
| Approach: | They propose to use a policy-document summarization dataset to enforce the summaries to include critical privacy-related entities and organization’s rationale in collecting those entities. |
| Outcome: | The proposed model improves over baselines and qualitatively evaluates the proposed model on human and qualitative data. |
Hybrid Alignment Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with instructions and preferences are conflicting . et al., 2023b) show that hybrid alignment training can outperform baselines . |
| Approach: | They propose a hybrid alignment training approach based on alternating alignment and modified elastic weight consolidation methods to achieve better collaboration between different alignment tasks. |
| Outcome: | The proposed approach outperforms baseline alignment training methods on summarization and dialogue tasks. |
CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models that generate harmful responses are constrained by their inherent capability. |
| Approach: | They propose to align large language models with human preferences from AI feedback. |
| Outcome: | The proposed framework improves the alignment of large language models with human preferences from AI feedback. |
Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have shown strong performance in tasks like radiology report generation but struggle with hallucinations, vague descriptions, Inconsistent logic and poor localization. |
| Approach: | They propose a framework for medical visual reasoning based on Visual Guidance and Self-Reward paradigms and Monte Carlo Tree Search to improve the model's visual reasoning capabilities. |
| Outcome: | The proposed framework outperforms existing models on multiple medical VQA benchmarks. |
PopALM: Popularity-Aligned Language Models for Social Media Trendy Response Prediction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work focuses on generic human responses without considering popularity factors in the social contexts. |
| Approach: | They propose Popularity-Aligned Language Models to distinguish responses liked by a larger audience through reinforcement learning. |
| Outcome: | The proposed model can distinguish responses liked by a larger audience through reinforcement learning. |
CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for proximal policy optimization discard valuable gradient signals from low-probability tokens due to the clipping mechanism. |
| Approach: | They propose an algorithm that reintroduces gradients from clipped tokens in native PPO in a gentle and bounded manner. |
| Outcome: | The proposed algorithm outperforms strong baselines on reasoning benchmarks on different model scales. |